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(54) MOBILE TERMINAL 

(57)Abstract: 

PROBLEM TO BE SOLVED: To provide a mobile phone 
capable of making speach communication between an 
audibly handicapped person and a normal hearing listener 
as well as between the audibly handicapped people, and 
creating a transmission text of electronic mail without 
the need for complicated key operations. 
SOLUTION: An image analysis section 8 of this terminal 
analyzes an image from a video camera input section 9, 
converts the image into character information and voice 
information corresponding to the image, and a display 
section 4 or a speaker 6 outputs the converted 
information. For example, the mobile terminal specifies 
uttered words from the motion and the shape of lips of a 
talker, the words are displayed, outputted from the 
speaker, and transmitted. Further, the mobile terminal is 
provided with a function that analyzes an image 
transmitted from an opposite party, converts the image 
into character information or voice information and 
outputs the converted information. Moreover, the 

converted output is used as control data for controlling the mobile terminal. 
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CLAIMS 



[Claim(s)] 
[Claim 1] 

the personal digital assistant equipment characterized by including an image-analysis means to 
be personal digital assistant equipment which has a video camera, and to analyze a motion of a 
speaker's lip obtained from said video camera, and the image of the configuration, and to change 
into text and (or) speech information. 
[Claim 2] 

the personal digital assistant equipment characterized by including an image-analysis means to 
be personal digital assistant equipment which has a video camera, and to analyze a speakers 
sign language image acquired from said video camera, and to change into text and (or) speech 
information. 
[Claim 3] 

Personal digital assistant equipment according to claim 1 or 2 characterized by registering 
beforehand the specified speaker which said image-analysis means analyzes. 
[Claim 4] 

the personal digital assistant equipment characterized by including an image-analysis means to 
analyze the alphabetic character image which is personal digital assistant equipment which has a 
video camera, and is acquired from said video camera, and to change into text and (or) speech 
information. 
[Claim 5] 

claims 1-4 characterized by including further an output means to output the text and (or) 
speech information which were obtained by said image-analysis means to a user — either — the 
personal digital assistant equipment of a publication. 
[Claim 6] 

claims 1-5 characterized by including further a transmitting means to transmit the text and (or) 
speech information which were obtained by said image-analysis means — either — the personal 
digital assistant equipment of a publication. 
[Claim 7] 

claims 1-6 characterized by said image-analysis means having further the function which 
analyzes the receiving image received from the outside and is changed into text and (or) speech 
information — either — the personal digital assistant equipment of a publication. 
[Claim 8] 

claims 1-7 characterized by controlling equipment based on the information acquired by said 
image-analysis means — either — the personal digital assistant equipment of a publication. 
[Claim 9] 

claims 1-8 characterized by including further the control key which makes activity control of 
said image-analysis means — either — the personal digital assistant equipment of a publication. 
[Claim 10] 

Said control key is personal digital assistant equipment according to claim 9 characterized by 
making activity control of said image-analysis means by operating it according to the procedure 
in which it is constituted so that the existing actuation key may be made to serve a double 



purpose, and said actuation key was able to be defined beforehand. 
[Claim 11] 

Said control key is personal digital assistant equipment according to claim 9 characterized by 
making it use together by the on-off operation of a light pen. 
[Claim 12] 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] 

Especially this invention relates to the personal digital assistant equipment which has a video 

camera about personal digital assistant equipment. 

[0002] 

[Description of the Prior Art] 

Multi-functionalization is progressing and the latest portable telephone can transmit now the 
image which carries a video camera and is acquired with this video camera as one of the 
function of that to a phase hand. Therefore, if a TV phone becomes possible and a hearing- 
impaired person also uses sign language by using this function, it can talk over the telephone. 
[0003] 

[Problem(s) to be Solved by the Invention] 

However, since those who can understand sign language are restricted very much except the 
sight-and-hearing-handicaps person, the message by the TV phone function using sign language 
is mostly restricted to hearing-impaired persons, and has the problem that anyone can do a 
message. 
[0004] 

Moreover, the electronic mail using a portable telephone has also spread widely, and by carrying 
out the key stroke of the contents of the mail, the user who transmits an electronic mail needs 
to do a sequential input 1 character at a time, and needs to create text. In this case, since, as 
for the portable telephone, small lightweight-ization is advanced, the class of actuation key is 
limited extremely. Therefore, in order to carry out an alphabetic character input, it will be 
required that it is required to operate one key repeatedly, very complicated actuation is forced 
upon the user, there should therefore be no key stroke and text should be generated simply. 
[0005] 

The purpose of this invention is offering the personal digital assistant equipment with which also 
enable the message with a hearing-impaired person and an auditory normal person, and anyone's 
enabled it to talk over the telephone not to mention hearing-impaired persons' message. 
[0006] 

Other purposes of this invention are offering the personal digital assistant equipment which a 

complicated key stroke's is lost and can generate text very simply. 

[0007] 

[Means for Solving the Problem] 

the personal digital assistant equipment by this invention is characterized by including an image- 
analysis means to analyze a motion of a speaker's lip obtained from a video camera, and the 
image of the configuration, and to change into text and (or) speech information. 
[0008] 

other personal digital assistant equipments by this invention are characterized by including an 
image-analysis means to analyze a speaker's sign language image acquired from a video camera, 
and to change into text and (or) speech information. 
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[0009] 

the personal digital assistant equipment of further others by this invention is characterized by 
including an image-analysis means to analyze the alphabetic character image acquired from a 
video camera, and to change into text and (or) speech information. 
[0010] 

furthermore, it is characterized by including a transmitting means to transmit the text and (or) 
speech information which were characterized by including an output means to output the text 
and (or) speech information which were obtained by said image-analysis means to a user, and 
were obtained by said image-analysis means. 
[0011] 

Moreover, it is characterized by controlling equipment based on the information acquired by said 

image-analysis means. 

[0012] 

Furthermore, it is characterized by including the control key which makes activity control of said 
image-analysis means, and said control key is constituted so that the existing actuation key may 
be made to serve a double purpose, and it is characterized by making activity control of said 
image-analysis means by operating it according to the procedure in which said actuation key was 
able to be defined beforehand. Moreover, it is characterized by using said control key together 
by the on-off operation of a light pen. 
[0013] 

furthermore, said image-analysis means is characterized by having the function which analyzes 
the receiving image received from, the outside and is changed into text and (or) speech 
information. 
[0014] 

An operation of this invention is described, the image from a video camera is analyzed, and it 
changes into an alphabetic character and (or) speech information, and enables it to output For 
example, the language and the sentence which the image which prepared the function 
(labiomancy) to analyze the emitted language, the pattern recognition function of sign language, a 
handwriting recognition function, etc., and was obtained from a motion and configuration of a 
speakers lip with the video camera means are changed into text or speech information, and are 
outputted and displayed on them, and it is made to transmit again. 
[0015] 

Moreover, based on the text and speech information which were obtained by the function to 
analyze language, such as this, the pattern recognition function of sign language, and the 
handwriting recognition function, the control data for the motion control of personal digital 
assistant equipment is generated, and various control can be performed. Furthermore, the image 
received from the outside is analyzed and it can change into text or speech information. 
[0016] 

[Embodiment of the Invention] 

Below, a drawing is used and explained about the example of this invention. Drawing 1 is the 
outline functional block diagram of the 1st example of this invention. In drawing 1 , the portable 
telephone by this example is the configuration of having an antenna 1, the wireless circuit 2, a 
control circuit 3, a display 4, the image-analysis control key 5, a loudspeaker 6, a microphone 7, 
the image-analysis section 8, and the camera (video camera) input section 9. 
[0017] 

Activity control of the analysis processing in the image-analysis section 8 is carried out for ** 
by which a speaker's image inputted from the camera input section 9 by the video camera is 
analyzed in the image-analysis section 8 by press actuation of the image-analysis control key 5. 
This image-analysis section 8 has the function (labiomancy function) to analyze the emitted 
language, from a motion and configuration of a speaker's lip, and has the function which changes 
and outputs that analysis result to text or speech information. 
[0018] 

In this way, the text and speech information which were changed are displayed on a display 4, 
and are outputted by the loudspeaker 6 as voice. In this case, both the display of an alphabetic 
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character and an audio output are good in a line, and only its one side is good. Moreover, based 
on the text and speech information which were changed, it changes into the control data (control 
command) in a control circuit 3, and various motion control of a portable telephone can be 
performed. 
[0019] 

Drawing 2 is a flow chart which shows actuation of the example of this invention. First, press 
actuation of the image-analysis control key 5 is answered (step S1), and the image-analysis 
section 8 will be in an active state, analyzes the input image from the camera input section 9, 
and specifies language (step S2). The language specified in the image-analysis section 8 is 
changed into text, and is outputted and displayed on a display 4 through a control circuit 3, 
Moreover, it is changed into speech information and outputted to a loudspeaker 6 through a 
control circuit 3. Furthermore, it is outputted as control data used in a control circuit 3 (step 
S3). 
[0020] 

Although the image-analysis section 8 will be in an inactive condition and it will return to an 
initial state if press actuation of the image-analysis control key 5 is made (step S4), if there is 
no press actuation, image-analysis processing will be continued. In addition, at step SI, if there 
is no press actuation of an image-analysis control key, of course, it will be maintained by the 
normal state (step S5). 
[0021] 

In step S3, in order to output as control data, it is necessary to identify whether the print-outs 
from the image-analysis section 8 are an alphabetic character/speech information, and whether 
it is control data. Therefore, the image currently analyzed in the current image analysis section 8 
adds the code for discernment which shows that it is control data to the output of the image 
analysis section 8. An operator (user) can perform addition of this code for discernment by the 
approach of pressing the specific key defined beforehand. 
[0022] 

The 2nd example by this invention is described. Although the configuration of this example is the 
same as that of drawing 1 , the function of the image-analysis section 8 is performing the 
pattern recognition of sign language. That is, a speaker's sign language pattern inputted from the 
camera input section 9 by the video camera is recognized, and the text corresponding to it, 
speech information, and also control data are generated. 
[0023] 

Although actuation of this example is also the same as that of the flow of drawing 2 , it is the 
image-analysis step S2, and sign language pattern recognition processing is performed and other 
actuation is the same as the 1st previous example. 
[0024] 

The 3rd example of this invention is described. Although the configuration of this example is the 
same as that of drawing 1 , the function of the image-analysis section 8 is performing character 
recognition. That is, the handwriting alphabetic character pattern inputted from the camera input 
section 9 by the video camera is recognized, and the text corresponding to it, speech 
information, and also control data are generated. 
[0025] 

Although actuation of this example is also the same as that of the flow of drawing 2 , it is the 
image-analysis step S2, and handwriting recognition processing is performed and other actuation 
is the same as the 1st example. 
[0026] 

moreover, above-mentioned the 1- in each 3rd example, each pattern of the user who should 
register the image analysis of a motion and configuration of a lip, sign language, and a handwriting 
alphabetic character so that only a specified speaker (user) may be possible can also be 
registered beforehand. 
[0027] 

furthermore, above-mentioned the 1- in each 3rd example, the data which were analyzed in the 
image-analysis section 8 and changed into text or speech information can be used also as 



transmit data, and serve as ready-for-sending ability through the wireless circuit 2. Furthermore, 
analysis processing of the image information received from the partner can be carried out in the 
image-analysis section 8 again, and it can change into text or speech information, and can also 
output to a display 4 or a loudspeaker 6. 
[0028] 

By carrying out like this, the message with not only hearing-impaired persons but a hearing- 
impaired person and an auditory normal person is attained. Moreover, since a message is 
possible and the contents of the message are not known by others even in a place where the . 
message manners in the location where many men exist are required since the message in an 
alphabetic character is attained silent, secrecy nature also improves. Furthermore, a complicated 
key stroke will not be needed for the creation time of the transmitted wording of a telegram for 
electronic mails, either, but it will be good for it at easy actuation again. 
[0029] 

Drawing 3 is the block diagram showing the 4th example of this invention, and the same sign 
shows drawing 1 and an equivalent part. In this example, it is the configuration of using the key 
(the existing key) of the key input section 5, instead of the image-analysis control key 5 of 
drawing 1 . For example, he is trying to use the on hook and ofHiook key of a portable telephone 
for the key for activity control of image analysis 8. By carrying out like this, it is effective in 
extension of a key becoming unnecessary. 
[0030] 

It is the flow Fig. showing actuation of this example, and drawing 4 answers press actuation of a 
key (carbon button) on hook (step SI), and analysis processing of the image-analysis section 8 
is activated, and it is started. Press actuation of an off-hook key (carbon button) will be 
answered (step S4), the analysis processing concerned will be deactivated, and it will end. Other 
actuation is the same as it of drawing 2 . 
[0031] 

Although the case where it was used, existing key, for example, on-hook/off-hook key, was 
explained instead of the image-analysis control key 5, you may replace with the image-analysis 
control key 5 by operating not only it but other existing keys (plurality or a single being 
sufficient) according to the procedure which was able to be defined beforehand. 
[0032] 

Next, the case where the light pen for a handwriting alphabetic character is used is explained 
instead of the image-analysis control key shown in drawing 1 , In the 3rd previous example, 
although the image-analysis section 8 performed the pattern recognition of a handwriting 
alphabetic character, a handwriting alphabetic character shall be drawn using a light pen at this 
time. Then, ON/OFF of this light pen are made to perform activity control of the image-analysis 
section 8. 
[0033] 

Drawing 5 is a flow Fig. of operation in this case. When a light pen is ON (step S1), it activates, 
and the image-analysis section 8 analyzes the input image from the camera input section 9, and 
changes it into the text and speech information according to a handwriting alphabetic character 
(step S3). If a light pen becomes off (step S4), it will return to the first condition. 
[0034] 

In each example mentioned above, the configuration and processing of the image-analysis 
section 8 may process by passing the result after sharing DSP (Digital Signal Processor) and 
CPU (Central Processing Unit) which are contained in the control circuit 3 of the conventional 
portable telephone and performing image analysis by DSP to CPU, and may be performed by 
software only by CPU. Moreover, although the portable telephone was described, it is clear that 
it is applicable similarly about the information processing terminal of a pocket mold. 
[0035] 

[Effect of the Invention] 

As stated above, according to this invention, as well as hearing-impaired persons' message, the. 
message with a hearing-impaired person and an auditory normal person can be performed, and it 
is effective in the message of anyone being attained. Moreover, there are creation of the 



communication link wording of a telegram for electronic mails and effectiveness of it becoming 
unnecessary to perform a complicated key stroke. Furthermore, since the control data of the 
motion control of equipment is also generable without a key stroke, it is effective in the ease of 
actuation and reduction of keys being attained. 
[0036] 

Furthermore, since it is not necessary to use an audio circuit network, without interrupting it 
while being able to do the activity which generates the control data which is control command, 
there is effectiveness that the circuitry of a message system can be simplified also during a 
message and data communication again. 
[Brief Description of the Drawings] 

[Drawing 1] It is the outline block diagram of one example of this invention. 

[Drawing 2] It is the flow Fig. of the example of drawing 1 of operation. 

[Drawing 3] It is the outline block diagram of other examples of this invention. 

[Drawing 4] It is the flow Fig. of the example of drawing 3 of operation. 

[Drawing 5] It is the flow Fig. of the example of further others of this invention of operation. 

[Description of Notations] 

1 Antenna 

2 Wireless Circuit 

3 Control Circuit 

5 Image-Analysis Control Key (Key Input Section) 

6 Loudspeaker 

7 Microphone 

8 Image-Analysis Section 

9 Camera Input Section 
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precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is the outline block diagram of one example of this invention. 

[Drawing 2] It is the flow Fig. of the example of drawing 1 of operation. 

[Drawing 3] It is the outline block diagram of other examples of this invention. 

[Drawing 4] It is the flow Fig. of the example of drawing 3 of operation. 

[Drawing 5] It is the flow Fig. of the example of further others of this invention of operation. 

[Description of Notations] 

1 Antenna 

2 Wireless Circuit 

3 Control Circuit 

5 Image-Analysis Control Key (Key Input Section) 

6 Loudspeaker 

7 Microphone 

8 Image-Analysis Section 

9 Camera Input Section 

[Translation done.] 
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